A guide to numeric measures

Computer programs are increasingly capable of grouping together expressions of a discourse that denote the same entity. We regard the evaluation of this capacity as the comparison between a system’s response and the one expected by the evaluators. We outline a theoretical framework for reference (§ 1) and another one for evaluation (§ 2), then analyze three existing quality measures (§ 3.2– 3.4), one of which was used in the MUC evaluation campaign. We propose mainly two new measures, one based on the notion of core equivalence class (§ 3.5), and the other based on information theory (§ 3.6), both showing better theoretical coherence than the previous ones. We also examine two alternatives, the exclusive core classes (§ 3.5.4) and the distributional measure (§ 3.7). In addition, we study a series of generalizations to the main problem (§ 4), and provide the results of all measures on several texts ( 5). Language-related tasks often require a certain degree of language understanding. Following a commonsense conception, understanding a linguistic message will be equated here with: (1) understanding what entities the message talks about; (2) understanding what the message says about these entities (their properties, relations, etc.) Our main goal is to estimate the capacity of a computer program to “understand references”, that is, to keep track of the various entities that a linguistic message is about. We will first describe a broad framework for this phenomenon, and situate the problem of coreference within it. 1. A FRAMEWORK FOR REFERENCE USE